Method and system for using combined voice and customized instructions to trigger vehicle reports

ABSTRACT

A method and system for an onboard reporting system capable of detecting keywords in voice data and considering the keywords in combination with respective predefined instructions is provided. The provided system uses, for each keyword, predefined instructions to validate the keyword prior to sending a report.

TECHNICAL FIELD

Embodiments of the subject matter described herein relate generally tovehicle report systems and, more particularly, to an aircraft reportsystem that uses combined voice and customized instructions to triggerthe sending of a report.

BACKGROUND

Increasingly, aircraft reports are used in incident analysis.Traditionally, incident analyses relied upon data from aircraftequipment, such as a Flight Data Recorder (FDR) and a Cockpit VoiceRecorder (CVR). The FDR generally records avionics data from variousonboard sensors and aircraft subsystems, and the CVR generally recordsthe cockpit voices during the flight. In the event of an incident, datafrom the FDR and CVR is often analyzed in an incident analysis toascertain the root cause of the incident. However, in order to use datafrom the FDR and CVR for incident analysis, the FDR and CVR have tofirst be found.

In some scenarios, it may be difficult and time consuming to locate theFDR and CVR. Even when aircraft incidents occur over ground, it can takeseveral days to locate the FDR and CVR. Moreover, after the FDR and CVRare located, incident analysis may incur a lot of time and effort,because the FDR and CVR each typically contain a large amount ofinformation. Therefore, aircraft reports are increasingly relied upon tomake up for these deficiencies.

Some current onboard systems generate aircraft reports in reliance on aninternal database which includes a variety of pre-defined instructions(herein, the instructions may also be referred to as logic) that arecreated to assist in incident analysis. The pre-defined instructionsgenerally poll the status of one or more aircraft subsystems or sensors.The pre-defined instructions may be created using any currentlyavailable script language, and stored in an onboard database. In such asystem, typically, when one of the pre-defined instructions is metduring the flight, the onboard system generates a report includingessential avionics data and sends the report to a ground controlstation, such as an Air Traffic Control (ATC) station.

However, the aforementioned systems are primarily reactionary. Ananticipatory input may improve the system. Cockpit voice (defined hereinas the voice in the cockpit, the communications among flight crew, thecommunications between air and ground, and the aural alarms from theavionics system), may provide the valuable anticipatory input. Addingcockpit voice (for example, by scanning voice data for safety keywords)to the considerations for sending reports may cover some additionalurgent scenarios that current pre-defined instructions alone cannotcover. Although keywords such “stall,” “fire,” “pull up”, etc. arehighly unlikely, a reporting system designed with an abundance ofcaution may look for these words in cockpit voice data in order toprovide anticipatory input to the system. An onboard system capable ofdetecting a situation of concern earlier could trigger the sending of areport including essential aircraft parameters to the ground earlier,which may enable prevention, or timelier and more accurate incidentanalysis. However, it is essential to validate that any keywordscaptured from cockpit voice really mean something of concern ishappening on the aircraft before triggering a report, or false alarmsmay occur.

Accordingly, an onboard reporting system capable of detecting keywordsin voice data and considering the keywords in combination withrespective predefined instructions is desirable. The desired systemuses, for each keyword, predefined instructions to validate the keywordprior to sending a report to a ground station. The desired onboardsystem would improve the timeliness, accuracy and availability of thereports, as well as improving overall aircraft safety.

BRIEF SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription section. This summary is not intended to identify keyfeatures or essential features of the claimed subject matter, nor is itintended to be used as an aid in determining the scope of the claimedsubject matter.

A method, executable by a processor, for sending a report is provided.The method comprises: receiving audio data from an audio input device;converting the audio data into voice data; obtaining keyword data from adatabase, wherein the keyword data comprises a plurality of keywords,each of the plurality of keywords associated with respectiveinstructions to validate the keyword; processing the voice data and thekeyword data to detect a first keyword in the voice data; when the firstkeyword is detected, performing the steps of (i) executing theassociated instructions to determine whether the first keyword isvalidated, and (ii) sending the report when the first keyword isvalidated.

A system for sending a report from an aircraft to a ground station, theaircraft comprising an audio input device configured to provide audiodata, is provided. The system comprises: a database of keyword data,wherein the keyword data comprises a first keyword and associatedinstructions to validate the first keyword; and a processor comprising aspeech recognition product, coupled to the audio input device, and thedatabase, the processor configured to (i) receive audio data, (ii)convert audio data into voice data, (iii) obtain keyword data, (iv)detect the first keyword in the voice data, and, in response todetecting the first keyword, (a) execute the associated instructions todetermine whether the first keyword is validated, and (b) send thereport to the ground station when the first keyword is validated.

Another method, executable by a processor, for sending a report from anaircraft to a ground station, is provided. The method comprises:receiving voice data from a speech recognition product; obtainingavionics data from an onboard sensor or aircraft subsystem; obtainingkeyword data from a database, wherein the keyword data comprise a firstkeyword and associated instructions to validate the first keyword;processing the voice data and the keyword data to detect the firstkeyword in the voice data; when the first keyword is detected,performing the steps of (i) processing the avionics data in accordancewith the associated instructions to determine whether the first keywordis validated, and (ii) sending the report when the first keyword isvalidated.

Other desirable features will become apparent from the followingdetailed description and the appended claims, taken in conjunction withthe accompanying drawings and this background.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the subject matter may be derived fromthe following detailed description taken in conjunction with theaccompanying drawings, wherein, like reference numerals denote likeelements, and:

FIG. 1 is a simplified block diagram of a report triggering system,according to an exemplary embodiment;

FIG. 2 is a flow diagram of an exemplary report triggering processsuitable for use with the system of FIG. 1 in accordance with one ormore embodiments; and

FIG. 3 is a flow diagram providing additional detail for the step ofexecuting the associated instructions (STEP 207 to STEP 209) of thereport triggering process 200.

DETAILED DESCRIPTION

The following Detailed Description is merely exemplary in nature and isnot intended to limit the embodiments of the subject matter or theapplication and uses of such embodiments. As used herein, the word“exemplary” means “serving as an example, instance, or illustration.”Any implementation described herein as exemplary is not necessarily tobe construed as preferred or advantageous over any otherimplementations. Furthermore, there is no intention to be bound by anyexpressed or implied theory presented in the preceding Technical Field,Background, Brief Summary or the following Detailed Description.

Techniques and technologies may be described herein in terms offunctional and/or logical block components and with reference tosymbolic representations of operations, processing tasks, and functionsthat may be performed by various computing components or devices.Operations, tasks, and functions are sometimes referred to as being aset of “instructions;” such instructions may be stored in memory or adatabase and then computer-executed, computerized, software-implemented,or computer-implemented. The instructions may also be converted intohardware using logic gates and/or a field programmable gate array(FPGA).

In practice, one or more processor devices can carry out the describedoperations, tasks, and functions by manipulating electrical signalsrepresenting data bits at memory locations in the system memory, as wellas other processing of signals. The memory locations where data bits aremaintained are physical locations that have particular electrical,magnetic, optical, or organic properties corresponding to the data bits.It should be appreciated that the various block components shown in thefigures may be realized by any number of hardware, software, and/orfirmware components configured to perform the specified functions. Forexample, an embodiment of a system or a component may employ variousintegrated circuit components, e.g., memory elements, digital signalprocessing elements, logic elements, look-up tables, or the like, whichmay carry out a variety of functions under the control of one or moremicroprocessors or other control devices.

The following descriptions may refer to elements or nodes or featuresbeing “coupled” together. As used herein, unless expressly statedotherwise, “coupled” means that one element/node/feature is directly orindirectly joined to (or directly or indirectly communicates with)another element/node/feature, and not necessarily mechanically. Thus,although the drawings may depict one exemplary arrangement of elements,additional intervening elements, devices, features, or components may bepresent in an embodiment of the depicted subject matter.

While not the subject of this invention, it is readily appreciated thatthe aircraft report (“report”) will include the keyword itself and therelevant avionics data, such as aircraft parameters, and the status ofaircraft sensors and subsystems. As a result, the exemplary embodimentsaves the effort and cost of locating and analyzing the data within FDRand CVR, and enables timelier and more accurate incident analysis.

FIG. 1 is a simplified block diagram of report triggering system 102,according to the embodiment. Communications management unit (CMU) 106,audio input 104, cockpit display 110, and avionics sources 108 are eachcoupled to report triggering system 102. The report triggering system102 may reside within or be coupled to an onboard maintenance system.

The report triggering system 102 comprises a processor 112, memory 114,and speech recognition product 116. A variety of currently availablespeech recognition products 116 may be used to receive audio input 104from the cockpit, and parse it into words, typically as a text steam(the text stream is referred to herein as “voice data”). In anembodiment, the speech recognition product 116 may reside external tothe report triggering system 102 and supply the voice data to the reporttriggering system 102. In another embodiment, the speech recognitionproduct 116 may reside within the report triggering system 102, sharingprocessor 112 operations and memory 114. Regardless of the location ofthe speech recognition product 116, it provides a text stream of words(the voice data) to the report triggering system 102 for comparison tostored keywords.

In practice, processor 112 may comprise, or be associated with, anysuitable number of individual microprocessors, flight control computers,navigational equipment, memories (such as memory 114), power supplies,storage devices, interface cards, and other standard components known inthe art. In this respect, the processor 112 may include or cooperatewith any number of software models, software programs (e.g., aircraftdisplay programs) or instructions designed to carry out the variousmethods, process tasks, calculations, and control/display functionsdescribed below. As mentioned, processor 112 may be included within anonboard maintenance system, or may reside separately.

Memory 114 stores keywords and associated instructions for confirmingthe validity of the respective keyword. For each keyword, the associatedinstructions are customized to validate that the keyword is actuallyhappening, and is not just a fragment of random conversation among thecrew. Advantageously, the exemplary embodiment ensures that if aconcerning keyword is detected, and the associated instructions are met,the onboard system is triggered to generate a report and send it to theground station, generally via Datalink.

The embodiments presented herein associate one or more keywords withrespective customized instructions for validating the keyword. For eachkeyword, the associated instructions are customized, such that when theinstructions are “met,” they validate the keyword. For example, if thekeyword is “stall,” the customized instructions poll avionics data, suchas the status of relevant aircraft sensors and aircraft subsystems. Whenthe associated instructions are met, it is validated that a stall isindeed underway. Accordingly, the relationship between each keyword andits associated customized instructions is unique, and storage of thekeyword plus its associated customized instructions may be referred toas, for example, a data structure or a lookup table.

CMU 106 may be any suitable device for transmitting an aircraft reportto a ground station. In an embodiment, the CMU 106 transmits each reportvia Datalink. The CMU 106 also receives and processes air trafficcontroller (ATC) commands for use in the report triggering system 102.

Image-generating devices suitable for use as cockpit display 110typically take the form of a primary flight display (PFD) and amulti-function display (MFD), and include various analog (e.g., cathoderay tube) and digital (e.g., liquid crystal, active matrix, plasma,etc.) display devices. In certain embodiments, cockpit display 110 mayassume the form of a Head-Down Display (HDD) or a Head-Up Display (HUD)included within a rotorcraft's Electronic Flight Instrument System(EFIS).

During operation of the report triggering system 102, the display 110renders a displayed image that provides navigational, terrain, andweather information, as in traditional aircraft displays. Additionally,the processor 112 may generate message data and may command the display110 to overlay the traditional displayed image with message dataassociated with the report triggering system 102, or with message dataindicative of a status of a progress of generating and/or sending areport by the report triggering system 102. The processor processes saidmessages and determines, based on a respective keyword, whether todisplay a message and/or a status of a report. For example, caution maydictate that the system not display messages associated with keywordsassociated with certain incidents, such as “hijack,”.

FIG. 2 is a flow diagram of an exemplary report triggering process 200suitable for use with the system of FIG. 1 in accordance with one ormore embodiments. In the embodiment, the processor 112 receives audiodata at STEP 202. Audio data is converted to voice data at STEP 204. Inother embodiments, the processor 112 may receive voice data from anexternally located speech recognition product 116. The processor obtainskeyword data from a database (in the embodiment, the keyword databaseresides in memory 114). The audio data and keyword data are continuallyprocessed by the report triggering system 102 to detect a first keywordamong words in the voice data. At STEP 206, a first keyword is detectedamong the various words in the voice data.

When a first keyword is detected (STEP 206) in the voice data, theprocessor 112 retrieves the respective associated instructions frommemory 114 (i.e., the instructions associated with the first keyword),and executes the associated instructions (STEP 208). Executing theassociated instructions comprises processing the avionics data that iscontinually received from avionics source 108 (for example, an onboardsensor or aircraft subsystem) in accordance with the associatedinstructions to determine the validity of the first keyword. More detailfor the step of executing the associated instructions (STEP 207 to STEP209) is provided in FIG. 3.

The first keyword is validated at STEP 210. Validating the first keywordadvantageously protects against false alarms that may occur in reportingsystems that trigger a report in response to voice data alone. At STEP212, an aircraft report is generated and sent. In some embodiments,message data associated with a validated keyword is also generated anddisplayed on display 110. In an embodiment, the aircraft report is sentto a ground station. In other embodiments, the report may be sent toother suitable destinations. According to the safety designconsiderations and status of a given host aircraft status, the reporttriggering process 200 may end, or may start over again.

FIG. 3 is a flow diagram providing additional detail for the step ofexecuting the associated instructions (STEP 207 to STEP 209) of thereport triggering process 200. In an embodiment, once a first keyword isdetected (STEP 207), the report triggering process 200 furtheridentifies the first keyword (STEP 302) as one of two different types:immediate impact keywords and continuous checking keywords. An exampleof each type of keyword follows.

Immediate impact keywords execute the associated instructions once (STEP304) and proceed to STEP 210 to determine whether the keyword isvalidated. In the embodiment, “stall” is an immediate impact keyword.For example, if the aircraft is in flight, and the pilots' statementsinclude the word “stall”, the speech recognition product 116 willpresent “stall” to the processor 112. The processor detects “stall” fromamong the voice data and retrieves the associated instructions frommemory 114. In the embodiment, the associated instructions for “stall”include a variety of avionics parameter checks, such as a speed check,an angle of attack (AOA) check, and the like. The processor 112, havingaccess to avionics data, will execute the instructions associated with“stall” to determine the validity of the “stall” keyword utterance. Onlywhen “stall” is validated (STEP 210), the onboard system generates andsends the report (STEP 212) which includes the related parameters, suchas speed, AOA, position, etc., and the “stall” keyword to the groundstation. Advantageously, the ground station may immediately begin tolocate the aircraft and identify the reason for the “stall.”

Continuous checking keywords repeat at least some of the associatedinstructions (STEP 306) until a predetermined condition is met. In thismanner, one or more aircraft parameters are monitored continuously untilthe predetermined condition occurs (STEP 308). The predeterminedcondition may be, for example, a passage of a predefined amount of time,a status of an aircraft sensor or subsystem, the aircraft landing, acombination of parameters, or the like. In this regard, STEP 210 maytake a relatively long time to return a decision. During this time inSTEP 210, the report triggering system 102 continues to receive audiodata and process audio data, to detect any potentially additionalkeywords (as described in connection with STEP 206). In the embodiment,“hijack” is a continuous checking keyword. The processor detects“hijack” from among the voice data at STEP 206 and retrieves theassociated instructions from memory 114. The processor 112, havingaccess to avionics data, will continuously repeat at least some of theinstructions associated with “hijack” until the predetermined conditionis met (STEP 308) as a prerequisite to determining the validity (STEP210) of the “hijack” keyword utterance.

Only when “hijack” is determined valid (STEP 210), the onboard systemgenerates and sends the report (STEP 212) which includes the relatedparameters, and the “hijack” keyword to the ground station. In the eventof a hijack, the perpetrators may not immediately change the flight pathor flight plan. Therefore, in the embodiment, the instructionsassociated with “hijack” may cause the processor to continuously monitoraircraft parameters associated with the flight path or flight plan. Achange to the flight plan may be the predetermined condition used (STEP308) to validate the keyword “hijack” (STEP 210) and trigger thegeneration and sending of the hijack report (STEP2 12). Advantageously,the report triggering system is not requesting pilot or crewconfirmation in this case. Therefore, the designated recipient, such asthe ground station, may be timely alerted to take action accordingly,without putting aircraft crew in danger of a required user inputconfirmation step.

In another continuous checking example, the source of the audio inputmay be the cockpit aural alarm. In this example, an aircraft is flyingtoward a mountain and the keyword “PULL UP” from the cockpit aural alarmis detected (STEP 206) by the report triggering system 102, but theaircraft has not responded after the passage of an amount of time (thepredetermined condition STEP 308). In this scenario, the reporttriggering system 102 may generate and send a report (STEP 212) whichincludes the related parameters, such as speed, position, altitude,etc., together with the “PULL UP” keyword to the ground station. Inresponse, the ground station may immediately locate the aircraft andidentify the reason for the event.

As previously stated, only when the predefined instructions are met, isthe associated keyword considered validated and the respective reportgenerated and sent to the designated recipient, such as the groundstation. As previously stated, the utterance of these keywords in anaircraft cockpit is highly unlikely, the present design is intended toexpand aircraft safety by including these keywords in an abundance ofcaution.

Thus, there has been provided an onboard reporting system capable ofdetecting keywords in voice data and considering the keywords incombination with respective predefined instructions. The provided systemuses, for each keyword, predefined instructions to validate the keywordprior to sending a report to a ground station. The provided onboardsystem improves the timeliness, accuracy and availability of thereports, as well as improving overall aircraft safety.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or embodiments described herein are not intended tolimit the scope, applicability, or configuration of the claimed subjectmatter in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the described embodiment or embodiments. It should beunderstood that various changes can be made in the function andarrangement of elements without departing from the scope defined by theclaims, which includes known equivalents and foreseeable equivalents atthe time of filing this patent application.

What is claimed is:
 1. A method, executable by a processor, for sendinga report, the method comprising: receiving audio data from an audioinput device; converting the audio data into voice data; obtainingkeyword data from a database, wherein the keyword data comprises aplurality of keywords, each of the plurality of keywords beingassociated with respective instructions to validate the keyword, whereinvalidating the keyword is defined as validating that the keyword ishappening; processing the voice data and the keyword data to detect afirst keyword in the voice data; when the first keyword is detected,performing the steps of (i) executing the associated instructions todetermine whether the first keyword is validated, and (ii) automaticallyand without user input, sending the report when the first keyword isvalidated.
 2. The method of claim 1, further comprising: obtainingavionics data from an onboard sensor or aircraft subsystem; andprocessing the avionics data in accordance with the associatedinstructions, and the step of sending the report comprises sendingavionics data related to the keyword.
 3. The method of claim 2, whereindetermining whether the first keyword is validated comprisescontinuously repeating at least some of the associated instructionsuntil a predetermined condition is met.
 4. The method of claim 3,wherein the predetermined condition comprises a status of an avionicssystem.
 5. The method of claim 3, wherein the predetermined conditioncomprises a passage of an amount of time.
 6. The method of claim 3,wherein the predetermined condition comprises completion of landing bythe aircraft.
 7. The method of claim 2, further comprising identifyingthe keyword as an immediate impact keyword or a continuous checkingkeyword.
 8. The method of claim 7, wherein, when the keyword isidentified as a continuous checking keyword, determining whether thekeyword is validated comprises continuously repeating at least some ofthe associated instructions until a predetermined condition is met. 9.The method of claim 8, wherein the predetermined condition comprises astatus of an avionics system.
 10. The method of claim 8, wherein thepredetermined condition comprises a passage of an amount of time. 11.The method of claim 8, wherein the predetermined condition comprisescompletion of landing by the aircraft.
 12. A system for sending a reportfrom an aircraft to a ground station, the aircraft comprising an audioinput device configured to provide audio data, the system comprising: adatabase of keyword data, wherein the keyword data comprises a firstkeyword and associated instructions to validate the first keyword,wherein validating the keyword is defined as validating that the keywordis happening; and a processor comprising a speech recognition product,coupled to the audio input device, and the database, the processorconfigured to (i) receive audio data, (ii) convert audio data into voicedata, (iii) obtain keyword data, (iv) detect the first keyword in thevoice data, and in response to detecting the first keyword, (a) executethe associated instructions to determine whether the first keyword isvalidated, and (b) automatically and without user input, send the reportto the ground station when the first keyword is validated.
 13. Thesystem of claim 12, wherein the first keyword is one of the setcomprising: hijack, stall, fire, ice, pull up, and altitude.
 14. Thesystem of claim 12, further comprising: a source of avionics data; andwherein the processor is further configured to process avionics data inaccordance with the associated instructions.
 15. The system of claim 14,wherein the source of avionics data comprises an onboard sensor or anaircraft subsystem.
 16. The system of claim 12, wherein the processor isfurther configured to identify the first keyword as an immediate impactkeyword or a continuous checking keyword.
 17. The system of claim 16,wherein the processor is further configured to continuously repeat atleast some of the associated instructions until a predeterminedcondition is met to determine whether the first keyword is validated,when the first keyword is identified as a continuous checking keyword.18. The system of claim 12, wherein the first keyword is one of aplurality of keywords and wherein the keyword database comprises each ofthe plurality of keywords organized such that each keyword of theplurality of keywords is associated with a respective set ofinstructions sufficient to validate the keyword. 19.-20. (canceled)